Grouping My Personal Spotify Library

I wanted to take a closer look at my personal spotify library and try to find any interesting groupings or clusters that would be usefull when creating playlist.

This analysis is focused on clustering and exploring the attributes behind the music I listen to. With these models, I hope to strategically and quickly create playlists with songs more similar to eachother based off songs in my current library.

The Data

The data is collected using a spotify API connection library in python. I saved the data into a .csv file and use pandas for handling.

Most of the features are numeric values that rate and measure certain attributes of each track. For example: 'Danceability (float) describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. The distribution of values for this feature look like this:Danceability distribution'.

Data exploration and actions taken for data cleaning

For data cleaning, I elimintated a few non-float varibles like 'href' and 'id' which wouldnt help the clustering algorithm. After all the numeric data were identified, one of the first exploration techniques I use to understand the data involves looking at the feature correlations. If I wasnt using PCA later on, I would consider the correlation between loudness and energy as 'high' and look at alternative feature engineering techniques.Next, I fed the data through a pipeline that systematically log transformed and scaled the data using MinMaxScaler and numpy.log1p().

Training

K-Means.

The first model I created was k-means with number of clusters set to 4. As expected the 4 playlists created were very large and not very useful but I found it interesting their sizes varied from 82-635.

Next I wanted to look at inertia (tightness of clusters) and try to create playlists with clusters of smaller inertia keeping in mind there are over 1,500 records. I found the inertia for 1 cluster all the way through 50 clusters. and plotted on a graph.

The inertia is lowest with the model that finds 50 clusters and the range of songs in the created playlist was 11-56. When I look at the playlists created I found similarities in the songs that were grouped together. I also found duplicates in a few of the playlists which gave me confidence in the clustering algorithm.

Mean-Shift.

The second model implemented was a mean-shift algorithm where the number of clusters was decided by the model and the parameter 'quantile' (0-1) was tuned to 0.1. The 10 playlists created were a little similar in size ranging from 82-247.

KPCA Dimensionality Reduction.

The last models I trained used data with fewer features and experimented with deminsionality reduction using Kernel PCA. I experimented with data that ranged from 1-10 features and viewed the outputed playlists to try and find any interesting playlist combinations. I was able to find a a lot of uniqe playlist that grouped songs with similar sounds together. My library is mostly hip hop and rap but the latest models were able to put groups of songs together that were not typical sound of the overall library.

Final model

The best model is K-Means for this application because the user will have the biggest control over playlist size and k-means gernerally groups songs in similar cluster sizes. After investigating the playlists, the k-means model was able to create reliable groups of songs that I would consider putting into my library for personal use.

Summary - Key Findings, Insights and Next Steps

I found skipping the log tranformation step made the k-means model more likely to pick out and place 'the beep test'(a unique running exercise saved as a song) into its own cluster of 1 song. Implementing the log transformation to the data caused a more smooth and rounded set of playlist but track was incorrectly grouped together with other songs while it felt more like an outlier.

I found it incouraging that multiple songs made by the same artists made it into the same playlist and outlier songs were picked out.

Next Steps

Sample Playlist:

Import data

Adressing Skewed Data

Columns with skew greater than 0.75

Scaling Features

K-Means

Which songs are closest to the center points of the 3 clusters?

Mean Shift

Pipeline Transformations

KPCA

long description of the data

duration_ms

int The duration of the track in milliseconds.

acousticness

float A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic. The distribution of values for this feature look like this:Acousticness distribution

danceability

float Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. The distribution of values for this feature look like this:Danceability distribution

energy

float Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy. The distribution of values for this feature look like this:Energy distribution

instrumentalness

float Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0. The distribution of values for this feature look like this:Instrumentalness distribution

liveness

float Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live. The distribution of values for this feature look like this:Liveness distribution

loudness

float The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db. The distribution of values for this feature look like this:Loudness distribution

speechiness

float Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks. The distribution of values for this feature look like this:Speechiness distribution

valence

float A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry). The distribution of values for this feature look like this:Valence distribution

tempo

float The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration. The distribution of values for this feature look like this:Tempo distribution